So this matches my intuition around how to put all of this together. While we have no way of testing for a regression, I'd be okay with running with something like this for now. The thing I'll note is that when @ahejlsberg tried something like this with the system prompt, he got slightly worse results.

That said, he guessed part of this was probably tied to the schema falling further and further behind in history. So maybe something to alleviate this would be to instead do:

• prompt_prelude (user and bot messages)
• System: "You are a ..., up to and including the schema, with instructions for how to format the JSON"
• User: (the user request)

Basically, move the system prompt immediately before the user request.

I don't understand how your test tools work

No worries. To most quickly fix the build:

cd TypeChat/python
pip install -e .[dev,examples]
pytest --snapshot-update
git add tests
git commit -m "Update snapshots."

It probably also makes sense to make the repair prompt a system message as well.

One of the things I tried in https://github.com/microsoft/TypeChat/tree/systemPromptPython (ea0e0d8) was to take the approach here after fixing some issues. As @ahejlsberg mentioned, OpenAI unfortunately doesn't do a great job when you switch to the system prompts. Trying things out on the coffee shop example input file, it generally got more "wrong".

One idea we could try is to expose the (very basic) logic we use to parse out JSON so that people can override translate and do whatever they want.

But otherwise, the only other approaches I could advise to implement something like #238 is to experiment with preambles to talk about past state.

Thanks for the feedback!

Preamble issues

My problem with the preamble is that it is prefixed to the prompt containing the schema, which makes the conversation a bit awkward, as it goes like this (ignoring system messages):

• user: draw three squares
• bot: {some JSON}
• user: make them bigger
• bot: {some JSON}
• user: Please translate user requests into JSON using this schema: {schema JSON}. Here is a user question: make them red

What I want is this (because my intuition tells me that it's a more natural dialog):

• user: Please translate user requests into JSON using this schema: {schema JSON}. Here is a user question: draw three squares
• bot: {some JSON}
• user: make them bigger
• bot: {some JSON}
• user: make them red

About system messages:

My (rudimentary) understanding of system messages is that it's meant to set a behavior or personality that's hard(er) to subvert by users trying to be clever -- e.g. the system prompt would tell the model not to be racist. In TypeChat's case, I think that at best the prompt that sets the schema should be a system prompt, but the repair prompts should probably just be user prompts.

The docs say a little bit about this. It makes me think that it's not worth experimenting too much with this in particular.

I also just noticed "JSON Mode" in the docs, which may or may not be useful for TypeChat.

@gvanrossum FWIW, a while back I did a multi-turn experiment where I didn't include previous replies in the prompt, but rather just repeated the request with the additional user input concatenated to the the previous user input (separated by newlines). So, your example would have a single user prompt section with text like:

Please translate user requests into JSON using this schema:
{schema JSON}.
Here is a user question:
draw three squares
make them bigger
make them red

It seemed to work quite well. Wonder if that would work in your case?

I also just noticed "JSON Mode" in the docs, which may or may not be useful for TypeChat.

The reason we aren't using JSON Mode is that there is no good way to know upfront if a model supports it, and, unfortunately, older models don't know to ignore it, but instead error on the request.

Also, not getting JSON back from requests hasn't really been an issue.